.:===-  Software  Engineering  Institute 
Carnegie  Mellon  University 


Enabling  Incremental  Iterative 
Development  at  Scale:  Quality  Attribute 
Refinement  and  Allocation  in  Practice 


Neil  Ernst 
Stephany  Bellomo 
Robert  L.  Nord 
Ipek  Ozkaya 

June  2015 


TECHNICAL  REPORT 

C  MU/S  El -201 5-TR-008 


Software  Solutions  Division 

Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


http://www.sei.cmu.edu 


Copyright  2015  Carnegie  Mellon  University 

This  material  is  based  upon  work  funded  and  supported  by  the  Department  of  Defense  under  Contract 
No.  FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software  Engineer¬ 
ing  Institute,  a  federally  funded  research  and  development  center. 

Any  opinions,  findings  and  conclusions  or  recommendations  expressed  in  this  material  are  those  of  the 
author(s)  and  do  not  necessarily  reflect  the  views  of  the  United  States  Department  of  Defense. 

References  herein  to  any  specific  commercial  product,  process,  or  service  by  trade  name,  trade  mark, 
manufacturer,  or  otherwise,  does  not  necessarily  constitute  or  imply  its  endorsement,  recommendation, 
or  favoring  by  Carnegie  Mellon  University  or  its  Software  Engineering  Institute. 

This  report  was  prepared  for  the 
SEI  Administrative  Agent 
AFLCMC/PZM 

20  Schilling  Circle,  Bldg  1305,  3rd  floor 
Hanscom  AFB,  MA01731-2125 

NO  WARRANTY.  THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING 
INSTITUTE  MATERIAL  IS  FURNISHED  ON  AN  “AS-IS”  BASIS.  CARNEGIE  MELLON 
UNIVERSITY  MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED, 
AS  TO  ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR 
PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM  USE  OF 
THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY  WARRANTY 
OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT,  TRADEMARK,  OR 
COPYRIGHT  INFRINGEMENT. 

This  material  has  been  approved  for  public  release  and  unlimited  distribution  except  as  restricted  be¬ 
low. 

Internal  use:*  Permission  to  reproduce  this  material  and  to  prepare  derivative  works  from  this  material 
for  internal  use  is  granted,  provided  the  copyright  and  “No  Warranty”  statements  are  included  with  all 
reproductions  and  derivative  works. 

External  use:*  This  material  may  be  reproduced  in  its  entirety,  without  modification,  and  freely  distrib¬ 
uted  in  written  or  electronic  form  without  requesting  formal  permission.  Permission  is  required  for  any 
other  external  and/or  commercial  use.  Requests  for  permission  should  be  directed  to  the  Software  En¬ 
gineering  Institute  at  permission@sei.cmu.edu. 

*  These  restrictions  do  not  apply  to  U.S.  government  entities. 

DM-0002248 


Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


Table  of  Contents 


Abstract  vii 

1  Introduction  to  Quality  Attribute  Requirements  1 

2  Coping  with  Quality  Attributes  in  Iterative  Development  2 

3  Refining  Quality  Attribute  Requirements  4 

3.1  Ratcheting  5 

3.2  Horizontal  Slicing  5 

3.3  Prototypes  and  Spikes  6 

3.4  Goal  Elaboration  6 

3.5  Empirical  Evaluation  6 

4  Allocating  Quality  Attributes  to  Iterations  7 

4.1  YAGNI  8 

4.2  Hardening  Sprints  8 

4.3  Iteration  Zero  8 

4.4  Rework  9 

4.5  Evolutionary/Runway  9 

4.6  Edge  Cases  10 

4.7  Dependencies  10 

4.8  Empirical  Evaluation  of  Allocation  Approaches  1 1 

4.9  Summary  12 

5  Using  Existing  Practices  to  Manage  Iterations  13 

5.1  Software  Analysis  Techniques  13 

5.2  Software  Life-Cycle  Methodology  15 

6  Related  Approaches  17 

7  Conclusion  18 

References  19 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY  i 

Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


List  of  Figures 


Figure  1 :  Architecture-Centric  Engineering  for  Improving  Software  Systems  2 

Figure  2:  Allocation  Process  Patterns  7 

Figure  3:  Allocation  Process  Patterns — Edge  Cases  10 

Figure  4:  Allocating  QARs  Across  Iterations  of  Development  1 1 

Figure  5:  Survey  Responses  to  Allocation  Approaches  1 1 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


List  of  Tables 


Table  1 :  Approaches  to  Refinement 


4 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


v 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


Abstract 


Lengthy  requirements,  design,  integration,  test,  and  assurance  cycles  delay  delivery,  resulting  in 
late  discovery  of  mismatched  assumptions  and  system-level  rework.  In  response,  development 
methods  that  enable  frequent  iterations  with  small  increments  of  functionality,  such  as  agile  prac¬ 
tices,  have  become  popular.  But  such  methods  de-emphasize  architectural  analysis;  they  assume 
the  emergence  or  existence  of  a  stable  architecture.  Yet  as  the  business  goals  and  context  evolve, 
the  architecture  must  also  change,  which  requires  allocating  increments  of  quality  attribute  re¬ 
quirements  to  iterations  along  with  other  business  capabilities.  Quality  attribute  requirements 
(also  called  nonfunctional  requirements)  are  hard  to  separate  into  smaller  increments  since  they 
often  crosscut  many  aspects  of  the  product.  As  a  result,  allocation  is  uneven  since  it  is  challenging 
to  decompose  them  and  understand  their  value.  Working  with  quality  attribute  requirements  in  an 
incremental  and  iterative  fashion  involves  solving  two  problems:  separating  high-level  require¬ 
ments  into  their  constituent  parts  and  allocating  them  to  iterations  to  fulfill  the  requirement.  Un¬ 
derpinning  both  problems  is  the  need  for  measurements  to  show  that  the  requirement  is  satisfied. 
This  report  describes  industry  principles  and  practices  used  to  smooth  the  development  of  busi¬ 
ness  capabilities  and  suggests  some  approaches  to  enabling  large-scale  iterative  development,  or 
“agile  at  scale.” 
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1  Introduction  to  Quality  Attribute  Requirements 


Lengthy  requirements,  design,  integration,  test,  and  assurance  cycles  delay  delivery,  resulting  in 
late  discovery  of  mismatched  assumptions  that,  in  turn,  result  in  system-level  rework.  In  response, 
software  development  teams  have  turned  to  methods  that  enable  frequent  iterations  with  small  in¬ 
crements  of  functionality,  such  as  agile  practices.  But  these  methods  are  light  on  architectural 
analysis;  they  assume  the  emergence  or  existence  of  a  stable  architecture.  Yet  as  the  business 
goals  and  context  evolve,  the  architecture  must  change  as  well,  which  requires  allocating  incre¬ 
ments  of  quality  attribute  requirements  to  iterations  along  with  other  business  capabilities. 

Quality  attribute  requirements  (QARs)  are  qualifications  of  the  functional  requirements  of  the 
overall  product  [Bass  2012].  The  overarching  aim  in  dealing  with  quality  attributes  is  to  ensure 
that  the  system  satisfies  the  criteria  of  interest  to  judge  the  quality  of  a  system’s  operation,  rather 
than  specific  behaviors.  For  example,  building  a  system  to  have  “good  performance”  means  un¬ 
derstanding  what  “performance”  is  and  what  “good”  means.  In  iterative  software  development 
processes  (such  as  Scrum),  a  requirement  such  as  “improve  performance”  carries  implications  for 
the  design  and  infrastructure  of  the  system  that  will  often  span  multiple  iterations.  Consequently, 
a  development  team  needs  to  refine  high-level  QARs  into  subparts  that  are  iteration  sized.  In  do¬ 
ing  so,  they  must  answer  two  questions: 

1 .  What  are  the  important  tangible  refinements  of  the  QAR,  and  how  do  they  relate  to  each 
other?  This  is  both  an  analysis  and  a  design  activity. 

2.  How  are  the  parts  allocated  to  iterations  in  the  software  development  process?  This  is  a  pro¬ 
cess  management  activity. 

Underpinning  each  question  is  the  concept  of  measurement — to  show  that  the  requirement  and  its 
cost-benefit  tradeoffs  are  satisfied. 

QARs,  also  known  as  nonfunctional  requirements,  are  particularly  hard  to  refine  into  smaller  in¬ 
crements  since,  by  their  nature,  they  tend  to  crosscut  many  aspects  of  the  product.  As  a  result,  al¬ 
location  is  uneven  since  it  is  challenging  to  break  QARs  apart  and  understand  their  value.  In  this 
report,  we  identify  common  practices  for  refining  and  allocating  software  development  work  that 
focus  on  QARs  in  iterative  (or  agile)  software  development  processes.  In  these  settings,  there  is  a 
tendency  to  focus  on  functional  deliverables  at  the  expense  of  QARs:  “Customers  often  focus  on 
core  functionality  and  ignore  NFRs  [QARs]  such  as  scalability,  maintainability,  portability, 
safety,  or  performance”  [Cao  2008].  As  a  result,  costly  rework  is  often  required  to  correctly  sat¬ 
isfy  the  QAR  [Brown  2011]. 

We  discuss  some  mechanisms  to  eliminate  rework  related  to  QARs.  First,  we  introduce  quality 
attribute-focused  software  development  work;  then  we  discuss  practices  described  in  the  existing 
literature  that  deal  with  implementing  QARs,  from  refining  QARs,  to  allocating  them  to  itera¬ 
tions,  to  using  existing  industry  practices  to  manage  iterations.  This  overview  offers  development 
teams  a  number  of  ways  to  overcome  challenges  to  large-scale  incremental  iterative  development, 
or  “agile  at  scale.” 
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2  Coping  with  Quality  Attributes  in  Iterative  Development 


Figure  1  shows  the  basic  concepts  of  a  software  development  process  in  which  the  role  of  QARs 
in  a  software  development  project  can  be  understood.  QARs  are  refined  from  business  goals  (on 
the  left),  an  architecture  is  proposed  to  satisfy  them,  the  implementation  tasks  (on  the  right)  are 
allocated,  and  then  the  system  is  implemented  based  on  conforming  to  the  architecture.  Over  time 
the  system  evolves  to  satisfy  the  business  goals. 


Figure  1:  Architecture-Centric  Engineering  for  Improving  Software  Systems 

In  this  report,  we  look  at  how  QARs  are  dealt  with  in  highly  iterative  development.  Almost  al¬ 
ways  this  means  refinement  and  allocation  are  never  fully  completed  since  the  architecture  and 
implementation  are  continuously  evolving.  We  must  return  to  the  business  goals  at  the  end  of 
each  iteration  to  reexamine  how  well  they  are  satisfied  by  what  we  learned  in  designing  and  im¬ 
plementing  the  solution.  This  process  suggests  a  need  to  view  software  design  as  a  series  of  evolv¬ 
ing  decisions  at  varying  levels  of  abstraction:  selecting  a  programming  language,  deciding  on  a 
reference  architecture  (e.g.,  n  tier),  using  frameworks  [Cervantes  2013],  understanding  modules, 
and  selecting  tactics  and  patterns  to  guide  implementation. 

The  advantages  of  using  an  evolutionary  approach  have  been  well  documented  [Breivold  2012]. 
When  we  talk  about  refining  QARs  or  allocating  them  to  iterations,  we  do  not  suggest  that  a  de¬ 
velopment  team  performs  either  task  only  once.  Projects  can  have  many  different  durations  for  it¬ 
erations  and  increment  planning,  depending  on  the  software  development  context,  ranging  from 
weeks  to  years.  At  one  end  of  the  spectrum,  allocation  may  involve  pulling  from  a  backlog  user 
stories  that  fit  the  iteration  length  (e.g.,  one-  or  two-week  sprints  in  Scrum).  At  the  other  end  of 
the  spectrum,  allocation  of  QARs  might  be  pulled  from  a  Statement  of  Work  or  contractual  agree- 


CMU/SEI-2015-TR-008  |  SOFTWARE  ENGINEERING  INSTITUTE  |  CARNEGIE  MELLON  UNIVERSITY 
Distribution  Statement  A:  Approved  for  Public  Release;  Distribution  is  Unlimited 


2 


ment  detailing  the  work  breakdown  structure  for  the  next  two  or  more  years.  Neither  scenario  im¬ 
plies  the  completion  of  architecting  activity  that  drives  the  allocation  of  QARs  before  develop¬ 
ment  starts,  although  the  latter  has  wrongly  been  assumed  to  imply  so. 

A  QAR  focus  implies  an  effort  to  improve  the  state  of  satisfaction  of  the  QAR.  For  example,  an 
iteration  may  be  heavily  weighted  toward  improving  performance  or  a  reengineering  activity  to 
change  web  frameworks  in  response  to  a  need  for  more  flexibility.  QARs  and  the  work  associated 
with  them  are  not  as  independent  as  features,  and  the  development  team  must  carefully  consider 
dependencies  when  packaging  the  pieces  into  units  that  must  be  allocated  together  or  sequenced 
over  iterations  and  releases  [Nord  2014,  Sethi  2009]. 

As  noted  by  Bachmann,  Bass,  and  Klein,  “the  key  for  design  is  characterizing  the  set  of  changes 
that  a  particular  system  will  be  subjected  to”  [Bachmann  2003].  Bachmann  and  colleagues  argue 
that  there  are  four  challenges  in  moving  from  QARs  to  design: 

1 .  precisely  specified  QARs 

2.  enumeration  of  design  approaches  that  achieve  QARs 

3.  linkage  between  the  QAR  and  the  design  fragments  that  achieve  it 

4.  a  method  for  composing  the  various  design  fragments  into  a  cohesive  whole 

We  note  two  additional  challenges.  One  challenge  is  to  correctly  specify  the  QAR  to  a  level  of  de¬ 
tail  that  is  achievable  in  some  limited  period  of  time  (  the  release  cadence)  and  to  find  design  frag¬ 
ments  that  can  be  completed  in  one  iteration  (where  an  iteration  is  a  segment  of  the  release 
cadence).  We  call  this  the  refinement  problem.  The  other  challenge  is  an  allocation  problem:  to 
correctly  allocate  design  fragments  to  iterations  to  optimize  the  relationship  between  cost  and 
value.  This  relationship  is  a  complex  one.  In  some  situations,  deferring  implementation  may  lead 
to  a  cost  of  delay,  such  as  failing  to  introduce  a  caching  system  to  support  a  holiday  promotion 
component  and  avoid  outages  due  to  increased  traffic  [Anderson  2012,  Slide  37].  In  other  situa¬ 
tions,  implementation  choices  made  to  meet  the  constraints  may  lead  to  costly  rework. 
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3  Refining  Quality  Attribute  Requirements 


QARs  are  often  provided  in  unstructured  and  unclear  ways.  QAR  refinement  is  the  process  of 
elaborating  and  decomposing  a  QAR  into  a  specifiable,  unambiguous  form  achievable  in  one  re¬ 
lease1  as  well  as  finding  design  fragments  that  will  accomplish  that  task.  In  the  agile  literature, 
this  is  also  known  as  slicing  or  sizing  and  is  typically  applied  to  user  stories.  In  the  requirements 
literature,  this  has  been  called  the  Twin  Peaks  model  [Nuseibeh  2001]  because  one  iteratively  pro¬ 
gresses  down  the  peaks  of  requirements  and  architecture. 

Refinement  of  QARs  ideally  results  in  a  unit  of  work  small  enough  to  be  testable,  small  enough  to 
fit  in  an  iteration,  and  useful  enough  to  produce  value.  Getting  to  the  appropriate  size  is  a  process 
of  analysis  and  synthesis,  separating  abstract  requirements  into  constituent  parts  so  they  and  their 
interrelationships  can  be  studied,  combining  constituent  parts  into  a  unified  entity  that  meets  the 
criteria.  It  is  a  design  activity:  “it  surfaces  our  ignorance  of  the  problem  domain”  [Khan  2014]. 
Design  support  for  any  one  quality  attribute  will  need  to  be  traded  off  against  design  support  for 
realizing  functional  requirements  and  other  QARs,  fixing  defects,  making  infrastructure  invest¬ 
ments,  and  so  on.  Identifying  measures  for  the  quality  attributes  of  interest  is  essential  input  in 
guiding  the  design  activity,  making  compromises  among  competing  concerns,  and  scoping  incre¬ 
mental  improvements. 

There  are  a  number  of  ways  to  size  requirements,  some  based  purely  on  analyzing  the  require¬ 
ment,  some  based  on  the  work  involved  in  satisfying  the  requirement,  and  some  based  on  a  mix¬ 
ture  of  analyzing  the  problem  and  possible  design  fragment  that  contribute  to  the  solution.  We 
have  highlighted  approaches  to  sizing  requirements  in  Table  1. 


Table  1:  Approaches  to  Refinement 


Approach 

Description 

Source 

Vagueness 

Break  down  vague  terminology  such  as  manage 

Green  2013 

Lawrence  2009 

And/Or  decomposi¬ 
tion 

Split  on  conjunctions  And  Or 

Anton  1996 

Green  2013 

Acceptance  or  test 
criteria 

Satisfy  one  criterion  per  slice 

Green  2013 

Workflow/use  case 
steps 

Use  one  slice  per  step;  frequently  seen  as  an  anti-pattern 

Adzic  2012 

Green  2013 

Lawrence  2009 
Verwijs  2013 

Business  rule 

Use  one  slice  per  variation  in  a  rule 

Cervantes  2013 
Verwijs  2013 

Dependencies 

Use  one  slice  per  dependency 

Denne  2003 

Lawrence  2009 

User  interface  (Ul) 
alternatives 

Classify  by  input  (e.g.,  keyboard  vs.  mouse  selection)  or  output  (e.g., 
screen  size) 

Lawrence  2009 
Verwijs  2013 

A  release  is  delivery  of  the  software  to  the  broad  customer  base;  an  iteration  is  a  sequence  of  time  during  deliv¬ 
ery,  which  often  produces  working  software  that  is  not  necessarily  widely  released. 
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Approach 

Description 

Source 

Ratcheting 

Ratchet  quality  attribute  criteria,  such  as  “works”  vs.  “works  fast” 

Gilb  2007 

Humble  2010 

Kua  2013 

Lawrence  2009 

Wirfs-Brock  2011 

Prototype/spike 

Add  spikes  where  steps  are  unknown;  “investigate”  vs.  “implement” 

Lawrence  2009 

Horizontal  slicing 

Slice  according  to  architectural  layers  (various);  commonly  seen  as 
an  anti-pattern  (cake  metaphor)  since  it  silos  the  work 

Verwijs  2013 

Path  based 

Split  by  normal/happy  and  problem  paths  through  the  application 

Verwijs  2013 

Parameters 

Use  input  parameter  options  such  as  phone  number,  zip  code,  etc. 

Verwijs  2013 

Operation  type 

Classify  by  Create,  Read,  Update,  Delete,  etc. 

Verwijs  2013 

Roles 

Classify  by  user  vs.  admin 

Verwijs  2013 

Use  case 

Set  the  overall  parameters  via  use  cases;  slice  the  use  cases  with 
user  stories 

Cockburn  2008 

Hamburger  slicing 

Create  horizontal  slices  that  map  steps  in  use  cases;  then  extend 
vertically  according  to  improved  quality  criteria 

Adzic  2012 

Not  all  of  these  approaches  require  lengthy  elaboration,  but  a  few  deserve  more  detailed  explana¬ 
tion. 

3.1  Ratcheting 

The  techniques  of  ratcheting  and  acceptance  or  test  criteria  use  quantifiable  outputs  of  each  itera¬ 
tion  to  identify  new  opportunities  for  work.  For  example,  Iteration  1  asks  for  the  system  to  work. 
Iteration  2  for  the  system  to  be  faster,  and  Iteration  3  for  it  to  be  as  fast  as  possible.  There  is  a  dan¬ 
ger  that  over-operationalizing  might  lead  to  poor  comparisons:  improving  a  system  to  be  twice  as 
fast  might  involve  much  more  than  twice  the  work.  Methods  such  as  set-based  design  [Kennedy 
2014],  Planguage  [Gilb  2007],  and  landing  zones  [Wirfs-Brock  2011]  all  leverage  this  idea  of  pro¬ 
gressively  ratcheting  user  story  targets  to  improve  quality  attribute  response.  A  more  elaborate 
form  of  ratcheting  may  involve  the  other  components  of  a  user  story  or  quality  attribute  scenario, 
for  example,  changing  the  source  of  a  request  from  internal  to  external  actors. 

3.2  Horizontal  Slicing 

The  horizontal  slicing  and  workflow/use  case  techniques  amount  to  identifying  the  steps  of  the 
use  case,  scenario,  or  user  story  and  assigning  each  phase  to  a  requirement.  For  example,  the  login 
screen,  user  validation,  database  query,  and  business  logic  might  all  be  done  one  after  the  other. 
However,  many  think  this  is  counterproductive.  For  one,  it  prevents  end-to-end  testing  from 
working.  Walking  skeleton  or  “tracer  bullet”  approaches  [Basili  1975,  Cockburn  1996,  Hunt 
1999],  in  which  one  builds  a  very  simple  application  demonstrating  complete  (if  simplistic)  func¬ 
tionality,  at  least  allow  for  proper  tests.  Furthermore,  focusing  on  one  horizontal  slice  (such  as  the 
UI)  can  lead  to  premature  optimization  (for  example,  if  the  database  layer  changes  afterward). 

Approaches  including  hamburger  slicing  [Adzic  2012],  splitting  by  operation  type,  and  business 
rule,  on  the  other  hand,  allow  one  to  show  how  the  software  system  can  support  the  entire  sce¬ 
nario. 
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3.3  Prototypes  and  Spikes 


One  motivation  for  adopting  an  incremental  approach  is  to  favor  “responding  to  change  over  fol¬ 
lowing  a  plan”  [Beck  2001].  Prototypes  and  spikes  are  information-gathering  activities  that  un¬ 
cover  unknowns  about  a  design  problem  [Leffingwell  2011b].  After  all,  we  can  implement  only 
what  we  know.  Use  of  prototypes  or  spikes  might  be  triggered  if  the  project  has  mounting  tech¬ 
nical  debt  or  wide  variation  in  cost  and  effort  estimates.  In  some  cases,  a  spike  becomes  a  specific 
requirement,  and  “learn  more  about  X”  is  assigned  to  a  developer  or  architecture  team.  The  spike 
might  take  the  form  of  an  experiment  or  design  expansion,  for  instance,  by  using  A/B  testing.  One 
approach  is  to  do  prototyping  with  a  specific  QAR  focus,  as  described  by  Bellomo  and  colleagues 
[Bellomo  2013].  Typically  prototype  work  does  not  lead  to  code  that  is  released. 

3.4  Goal  Elaboration 

One  can  use  techniques  such  as  the  Goal-Based  Requirements  Analysis  Method  (GBRAM)  to  per¬ 
form  refinement  on  higher  level  QARs  using  And/Or  slicing  [Anton  1996].  Developers  begin  top- 
down  refinement  by  asking  how  the  top-level  goal  might  be  achieved  in  some  combination  of  sub¬ 
goals.  This  approach  lends  itself  to  formal  analysis,  and  modeling  and  development  teams  often 
prefer  it  in  settings  where  detailed  safety  or  hazard  analysis  is  desired.  For  example,  search  algo¬ 
rithms  can  recommend  optimal  (e.g.,  minimal)  solutions  to  the  top-level  goals.  In  a  variation  on 
this  approach,  Gottesdiener  and  Gorman  break  high-level  or  abstract  constraints  along  six  dimen¬ 
sions  for  expansion:  user,  actions,  data,  business  rules,  interfaces,  and  quality  attributes 
[Gottesdiener  2010].  This  creates  a  much  larger  yet  better  described  epic.  In  particular,  they  ad¬ 
vise  exploring  options  for  common  quality  attributes  (security,  performance,  etc.)  and  then  using 
Planguage  tags  to  anchor  the  expanded  user  story  to  a  measurable  outcome  [Gilb  2007]. 

3.5  Empirical  Evaluation 

In  a  related  paper,  we  described  two  case  studies  that  we  conducted  with  software  companies 
[Bellomo  2014b;  Ernst  2014].  We  investigated  how  these  firms  managed  architectural  work  and 
found  that  ratcheting  was  the  primary  approach.  We  observed  that  developers  refined  performance 
requirements  using  a  feedback-driven  approach,  which  allowed  them  to  parse  the  evolving  perfor¬ 
mance  requirements,  expressed  as  state  transitions,  to  meet  increasing  user  expectations  over  time. 
Within  each  state  transition,  developers  refined  crosscutting  concerns  into  requirements  by  break¬ 
ing  them  into  their  constituent  parts  in  terms  of  the  scope  of  the  system  and  response  to  stimuli  in 
a  given  context.  The  system  and  crosscutting  performance  requirements  evolve  as  the  stimuli, 
context,  and  response  are  ratcheted. 

We  see  evidence  of  projects  that  are  better  able  to  sustain  their  development  cadence  with  a  com¬ 
bination  of  refinement  and  allocation  techniques  guided  by  measures  for  requirement  satisfaction, 
value,  and  development  effort.  As  we  retrospectively  analyzed  these  examples,  we  found  that 
these  teams  did  not  follow  a  formal  technique;  however,  they  did  have  common  characteristics  in 
how  they  refined  the  work  into  smaller  chunks,  enabling  incremental  requirements  analysis  and 
allocation  of  work  into  implementation  increments. 
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4  Allocating  Quality  Attributes  to  Iterations 


The  purpose  of  sizing  work  is  to  allocate  pieces  of  work  to  iterations.  In  an  agile  process,  alloca¬ 
tion  involves  taking  stories  from  the  product  backlog  and  adding  architectural  stories  directly  to 
the  iteration  backlog  [Gottesdiener  2010].  This  does  not  mean  that  the  task  breakdown  is  planned 
months  in  advance.  The  backlog  contains  stories  that  one  would  classify  as  specific  to  a  particular 
quality  attribute  and  design  fragments  associated  with  the  quality  attribute.  Publically  available 
examples  of  stories  can  be  seen  in  industry-relevant  open  source  systems.  For  example,  the 
CONNECT  and  HADOOP  Distributed  File  System  (HDFS)  projects  have  many  user  stories  that 
deal  with  performance  metrics  and  documentation  [CONNECT  2014,  Apache  2014].  The  ad¬ 
vantage  is  that  architectural  slices  have  high  visibility;  the  disadvantage  is  that  these  stories  may 
be  poorly  thought  out  and  slip  off  the  backlog. 

Figure  2  illustrates  several  allocation  process  patterns.  These  patterns  summarize  how  different 
software  development  life  cycles  can  be  generalized  with  respect  to  how  they  balance  architecting 
with  feature  development.  We  describe  these  patterns  and  provide  key  questions  that  the  develop¬ 
ment  team  should  address  in  allocating  QARs  within  the  boundaries  of  that  pattern. 


Figure  2:  Allocation  Process  Patterns 

Solid  green  lines  indicate  feature/functional  work,  dashed  yellow  lines  architectural  work,  (a)  No  archi¬ 
tectural  work  (YAGNI);  (b)  hardening  sprints;  (c)  Iteration  Zero;  (d)  rework;  (e)  evolutionary/runway. 
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4.1  YAGNI 


In  Figure  2(a),  the  “You  Aren’t  Going  to  Need  It”  (YAGNI)  practice  ignores  any  architectural 
work  as  non-value  added,  which  is  why  the  pattern  image  has  no  architectural  line.  To  be  fair, 
YAGNI  does  not  mean  never  do  architectural  work  but  rather  evolve  and  refactor  it  [Seguin 
2012].  Pure  versions  of  this  method  are  more  evolutionary  (see  Section  4.5).  However,  in  practice 
it  has  been  interpreted  to  mean  “never  do  design.”  YAGNI  appears  to  be  a  popular  approach  be¬ 
cause  of  the  significant  upfront  savings  in  architecture  effort.  However,  taking  shortcuts  can  result 
in  significant  re-architecting  effort  and  rework  later  on  [Bellomo  2014a]. 


Key  Questions  Will  you  ever  “ need  it”?  How  does  the  development  effort  value  architectural 
work  during  the  current  increment ? 


4.2  Hardening  Sprints 

Traditionally,  a  hardening  sprint,  shown  in  Figure  2(b),  has  been  defined  as  the  final  sprint  prior 
to  release  that  is  dedicated  to  fixing  bugs  and  improving  QAR  satisfaction.  This  is  called  an  anti- 
pattern  since  these  bugs  should  have  been  fixed  as  part  of  the  work  that  introduced  them.  How¬ 
ever,  development  teams  considering  how  to  scale  agile  use  this  term  to  group  together  produc¬ 
tion-release  tasks  that  only  occur  immediately  prior  to  release,  such  as  in  Leffingwell’s  Scaled 
Agile  Framework  (SAFe)  [Leffingwell  2014b]  or  the  Disciplined  Agile  Delivery  method’s  Transi¬ 
tion  Phase  [Ambler  2012],  Leffingwell  also  uses  this  phase  for  verification  and  validation  activi¬ 
ties  in  high-assurance  environments  [Leffingwell  2011a],  You  can  see  an  example  in  CONNECT 
Sprint  120  [CONNECT  2013].  A  variation  of  this  pattern  is  called  cleanup,  in  which  rather  than 
cleaning  up  the  code  prior  to  release,  the  code  is  first  released  and  subsequent  architectural  work 
is  done  to  clean  up  the  code  base.  Galen  gives  an  overview  of  terminology,  including  release  read¬ 
iness,  stabilization,  or  spring  cleaning  sprints  [Galen  2014].  Dedicating  a  single  sprint  to  cleanup 
means  that  the  entire  team  is  engaged,  though  such  a  sprint  makes  a  good  target  for  cutting  if  time 
is  a  factor.  Architectural  issues  are  not  always  amenable  to  deferral. 


Key  Questions  How  often  are  hardening  sprints  needed?  How  many  are  enough? 


4.3  Iteration  Zero 

In  Figure  2(c),  Iteration  Zero  involves  architecture  planning  before  writing  any  code.  An  overly 
long  Iteration  Zero  is  equivalent  to  the  dysfunctional  “Big  Up-Front  Design”  (BUFD)  anti-pattern. 
Meta-methods  like  the  Rational  Unified  Process  [Leffingwell  2011b]  and  Disciplined  Agile  De¬ 
livery  [Ambler  2012]  include  a  preliminary  design  phase,  which  may  itself  be  iterative  and  which 
Kruchten  calls  “architectural  iterations”  [Kruchten  1998].  Logically  every  development  effort 
needs  at  least  some  degree  of  initial  envisioning  or  a  well-understood  platform  with  a  well-de¬ 
fined  architecture  to  start  from.  Architectural  Iteration  Zero  is  specifically  about  deciding  on  some 
important  architectural  properties,  including  frameworks  and  patterns.  Figure  2(c)  captures  this 
concept  in  the  length  of  time  before  the  amount  of  work  on  features  and  functionality  (solid  green 
line)  exceeds  the  work  done  on  architecture  and  planning  (dashed  yellow  line).  Variations  of  Iter¬ 
ation  Zero  capture  tradeoffs  between  over-analysis  and  rework,  best  illustrated  by  Boehm’ s 
“sweet  spot”  discussion  [Boehm  2003]. 
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Iteration  Zero  fits  with  well-established  approaches  to  planning  and  project  management,  particu¬ 
larly  in  government  contexts.  It  provides  an  opportunity  to  take  a  high-level  view  of  the  problem 
before  becoming  enmeshed  in  low-level  work.  However,  it  is  possible  to  succumb  to  analysis  pa¬ 
ralysis.  It  is  also  unlikely  that  a  development  team  can  understand  all  ramifications  of  decisions 
prior  to  feedback  from  experimentation  and  implementation.  While  Iteration  Zero  implies  an  ac¬ 
tivity  that  happens  before  the  rest  of  the  design  and  development  effort,  the  essence  of  the  ap¬ 
proach  is  the  focused  planning  and  design  activity.  A  large-scale  development  effort  may  have 
several  Iteration  Zeros  for  different  parts  of  the  system. 


Key  Questions  How  long  should  Iteration  Zero  be  (i.e.,  what  amount  of  design  is  necessary 
vs.  wasteful)?  When  should  Iteration  Zero  be  revisited  for  different  parts  of 
the  system  ? 


4.4  Rework 

As  shown  in  Figure  2(d),  rework  is  more  of  an  anti-pattern,  wherein  feature  development  comes 
to  a  screeching  halt  as  technical  debt  becomes  impossible  to  ignore  [Nord  2012],  and  substantial 
portions  of  the  codebase  must  be  rebuilt.  Typically  this  approach  is  also  associated  with  substan¬ 
tial  rework  costs.  When  architecture  is  accounted  for  up  front,  it  can  allow  for  rapid  exploration  of 
alternatives  to  understand  and  manage  rework,  but  this  approach  costs  time  and  effort,  and  it  de¬ 
lays  other  feature  work. 


Key  Questions  When  are  evolutionary  architectural  improvements  still  possible  within  the 
release  cadence?  When  does  development  cross  the  boundary  that  necessi¬ 
tates  rewriting  the  system  (disrupting  development )? 


4.5  Evolutionary/Runway 

A  runway,  shown  in  Figure  2(e),  “exists  when  there  is  sufficient  system  infrastructure  in  place  to 
allow  incorporation  of  near-term  product  backlog”  [Leffingwell  2008].  It  emphasizes  a  low  level 
of  ongoing  architectural  work  that  comes  and  goes,  hence  the  wavy  pattern.  It  differs  from  agile 
approaches  in  which  user  stories  for  architecture  are  assigned  to  the  product  backlog  as  a  whole; 
runways  tend  to  be  separate  work  products  and  are  more  common  in  larger  projects. 

A  related  approach  is  vertical  slicing,  where  one  builds  out  the  runway  based  on  which  stories 
need  a  given  architectural  element  [Brown  201 1].  In  the  work  of  Denne  and  Cleland-Huang,  verti¬ 
cal  slicing  is  managed  with  a  dependency  graph:  one  first  decides  which  minimally  marketable 
features  to  prioritize  and  then  identifies  the  common  architectural  elements  to  those  high-priority 
features  [Denne  2003].  Properly  sequenced,  the  runway  team  can  act  in  coordination  with  devel¬ 
opment  to  evolve  the  architecture  needs.  However,  a  runway  team  may  potentially  lose  contact 
with  the  state  of  development  in  the  main  implementation  branch. 


Key  Questions  How  does  the  development  team  identify  opportune  moments  and  opportune 
places  for  minor  re-architecting  improvements?  How  long  should  the  runway 
be? 
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4.6  Edge  Cases 


The  final  patterns  are  better  described  as  edge  cases.  In  Figure  3(f),  both  feature  and  architecture 
work  increase  over  time,  as  one  might  see  as  a  product  scales  up  to  meet  increasing  demand.  In 
Figure  3(g),  the  opposite  holds,  for  example,  if  a  project  has  reached  the  end  of  its  life  cycle  or 
work  on  a  version  has  been  completed.  These  are  simple  and  easily  understood  patterns  that  often 
result  from  a  project  being  overcome  by  events.  However,  features  and  architecture  may  not  grow 
or  shrink  at  the  same  rate. 


Key  Question  At  what  point  do  the  levels  of  effort  start  to  change  ( i.e.,  what  is  the  inflection 
point)? 


Figure  3:  Allocation  Process  Patterns — Edge  Cases 
(f)  Both  architecture  and  feature  work  increase;  (g)  both  types  of  work  decrease. 


4.7  Dependencies 

The  process  patterns  presented  so  far  treat  quality  attributes  and  related  architecture  work  as  a  sin¬ 
gle  abstraction  to  be  allocated  with  other  work  during  software  development.  Here  we  switch  fo¬ 
cus  to  look  at  the  individual  quality  attributes  to  examine  the  information  needed  to  determine 
which  QARs  to  work  on  at  any  given  time  and  how  to  sequence  them  with  respect  to  each  other. 

In  the  SQALE  method,  Letouzey  proposes  a  dependency  hierarchy  for  QARs  that  suggests  a  se¬ 
quence  of  work  for  these  approaches  [Letouzey  2013]. 2  Testability,  reliability,  and  changeability 
are  the  qualities  of  highest  priority,  since  without  them  a  development  team  has  no  ability  to  work 
on  others.  (Note  that  testability  is  one  form  of  measurement.  It  is  not  about  the  ability  to  test;  it  is 
about  providing  the  data  that  shows  that  the  quality  attribute  properties  are  achieved.) 

In  Software  by  Numbers,  Denne  and  Cleland-Huang  use  dependencies  from  one  quality  attribute 
to  another,  and  from  quality  attributes  to  minimally  marketable  features,  to  understand  what  needs 
to  be  worked  on  and  when  [Denne  2003].  Goal  models  such  as  those  proposed  by  KAOS  are  an¬ 
other  dependency  mapping  approach  [Dardenne  2007]. 


SQALE  (Software  Quality  Assessment  based  on  Lifecycle  Expectations)  is  a  method  to  support  the  evaluation 
of  a  software  application  source  code.  It  is  a  generic  method,  independent  of  the  language  and  tools  for  source 
code  analysis. 
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In  Figure  4,  we  can  see  how  these  concepts  relate.  We  have  three  QARs  in  the  backlog.  We  have 
three  examples  of  how  the  QARs  were  allocated  across  the  first  three  iterations  of  development. 

In  the  first  example,  we  are  primarily  concerned  with  where  and  when  our  work  should  occur,  that 
is,  the  allocation  in  which  we  can  assign  QARs  to  iterations  without  further  refinement  or  decom¬ 
position.  In  the  second  example,  we  decompose  SI  into  constituent  parts.  Finally,  in  the  third  ex¬ 
ample,  our  concern  is  how  to  set  measurable  outcomes  on  our  progress  toward  meeting  SI,  an 
illustration  of  ratcheting. 


Quality  Attribute 
Requirements 


S3 

si 

S2 

SI' 

SI" 

SI'" 

si  M 

SI  [r  <  100]  SI  [r<  50] 

Iterations 


Figure  4:  Allocating  QARs  Across  Iterations  of  Development 


Key  Questions 


How  does  the  development  team  properly  value  the  nonfunctional  stories? 
Is  the  story  properly  sized  for  work  in  one  sprint? 


How  constrained  are  the  dependencies?  Is  it  possible  to  work  around  a  de¬ 
pendency? 


4.8  Empirical  Evaluation  of  Allocation  Approaches 

We  conducted  a  survey  of  three  large  organizations,  two  of  which  are  Fortune  500  organizations. 
Figure  5  shows  responses  to  a  question  that  presented  respondents  with  Figure  2  and  asked  them 
which  pattern  their  most  recent  project  best  matched.  Perhaps  unsurprisingly,  most  projects  did 
some  form  of  up-front  architecting.  Next  most  common  was  parallel  development,  but  a  substan¬ 
tial  number  conducted  agile  development. 


a)  Agile  development 
without  a  specific  focus  on 
architecting 

b)  Agile  development  with 
architecting  sprints  as 
needed 

c)  Primarily  up-front 
architecting  followed  by 
development 

d)  Initial  feature 
development  followed  by 
rearchitecting 


e)  Parallel  development  of 
features  and  architecture 

0%  5%  10%  15%  20%  25%  30%  35% 


Figure  5:  Survey  Responses  to  Allocation  Approaches 
Letters  match  choices  in  Figure  2. 
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4.9  Summary 


All  of  the  patterns  occur  in  practice,  although  some  patterns  occur  more  frequently  in  particular 
business  contexts.  For  example,  in  a  recent  survey  conducted  at  a  large  government  contractor,  we 
found  that  one-third  of  respondents  were  using  up-front  architecting  (Pattern  c),  yet  nearly  as 
many  were  engaged  in  parallel  architecting  and  development  activities  (Pattern  e).  Often  a  devel¬ 
opment  team  uses  up-front  architecting  when  there  is  a  lengthy  development  life  cycle,  a  safety- 
critical  system  context,  or  several  regulatory  bodies  that  must  be  involved  in  the  development  pro¬ 
cess. 

The  natural  way  these  patterns  are  used  is  in  combination.  For  example,  it  is  not  uncommon  to  see 
a  development  team  start  with  evolving  an  existing  system  (Pattern  e),  followed  by  Iteration  Zero 
to  conduct  some  architecture  planning  to  address  new  business  goals  (Pattern  c),  at  which  point 
the  team  puts  more  emphasis  on  feature  development  (Pattern  a)  and  conducts  no  further  architec¬ 
tural  work  until  a  need  to  introduce  new  QARs  arises,  when  the  effort  switches  to  rework  (Pattern 
d).  Similarly,  many  successful  agile  software  development  projects  start  with  an  architecture  run¬ 
way  (Pattern  e)  and  supplement  with  a  hardening  sprint  (Pattern  b)  [Bellomo  2013]. 
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5  Using  Existing  Practices  to  Manage  Iterations 


We  have  shown  that  part  of  design  in  an  iterative  setting  requires  understanding  how  to  refine  and 
allocate  QARs  and  functional  requirements  to  iterations.  This  section  offers  an  overview  of  how 
existing  software  analysis  techniques  and  life-cycle  planning  methods  can  help  a  development 
team  manage  these  iterations  to  enable  incremental  iterative  development  at  larger  scales.  Here  we 
summarize  requirements  elicitation  and  design-fragment  analysis  techniques  including  sprint 
planning,  INVEST  criteria,  Planguage,  set-based  design,  quality  attribute  elicitation  models,  and 
cost-benefit  analysis  methods.  Then  we  review  several  software  life -cycle  methods  including 
SAFe,  disciplined  agile  delivery,  and  the  incremental  funding  method.  Using  some  combination 
of  these  techniques  and  methods,  as  appropriate  to  the  software  development  effort,  will  help  a 
development  team  employ  architecture  in  service  of  smoothing  the  development  of  business  capa¬ 
bilities. 

5.1  Software  Analysis  Techniques 

User  Stories 

The  product  backlog  holds  a  collection  of  user  stories,  which  are  requirements  expressed  using 
(conventionally)  the  form  “As  a  <user>,  I  would  like  to  <activity>,  in  order  to  <goal>”  [Cohn 
2004].  The  product  owner  and  development  team  select  stories  according  to  immediate  value. 
Backlog  refinement  is  an  ongoing  process  that  ensures  the  stories  in  the  backlog  are  appropriately 
refined  and  properly  sized.  Allocation  happens  immediately  prior  to  the  sprint.  Wirfs-Brock  uses 
“landing  zones”  to  ensure  that  the  development  team  has  some  flexibility  in  making  tradeoffs 
among  requirements  [Wirfs-Brock  2011].  Each  requirement  has  a  range  of  acceptable  values  la¬ 
beled  minimum,  target,  or  outstanding.  A  landing  zone  is  similar  to  release  criteria  and  allows  for 
tolerances  in  acceptable  values.  Bjarnason  and  colleagues  reported  that  for  requirements,  iterative 
development  helped  prevent  overscoping  [Bjarnason  2011]. 

INVEST  Criteria 

The  INVEST  criteria  are  that  a  requirement  be  “Independent,  Negotiable,  Valuable,  Estimable, 
Small,  and  Testable”  [Lawrence  2009].  These  criteria  are  commonly  used  to  evaluate  the  quality 
of  a  given  user  story,  but  clearly  they  map  well  to  the  notion  of  allocating  and  refining  require¬ 
ments.  Independence,  negotiability,  and  valuation  are  all  linked  to  allocation,  and  estimableness 
and  smallness  are  refinement  guidelines.  Testability  is  a  value  judgment  to  measure  success.  The 
criteria  are  subjective,  so  understanding  definitions  (e.g.,  how  small?)  in  a  given  context  is  vital. 
However,  it  is  not  clear  that  these  criteria  are  sufficient  for  good  quality  attribute  requirements,  as 
they  were  applied  to  functional  user  stories  first. 

Planguage 

With  Planguage,  Gilb  defines  tags  for  architectural  objectives  (most  often  QARs)  [Gilb  2007]. 
Planguage  uses  Scale,  Meter,  Minimum,  Plan,  and  Wish  tags  to  standardize  stories.  For  example, 
for  the  response  time  component  of  a  performance  quality  attribute,  our  scale  is  seconds,  the  me¬ 
ter  is  time  between  the  user  pressing  a  button  and  an  outcome,  the  minimum  acceptable  outcome 
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is  7  seconds,  the  plan  is  for  4  seconds,  and  the  wish  is  for  2  seconds.  These  tags  provide  a  refine¬ 
ment  approach  to  implementing  the  software.  In  other  words,  a  development  team  can  first  ensure 
that  the  implementation  will  satisfy  the  minimum  outcome,  and  in  later  iterations  they  can  focus 
on  improving  response  time  to  the  plan  or  wish  levels.  Operationally,  the  team  can  use  the  Plan- 
guage  technique  to  create  independent  requirements  for  allocation. 

Set-Based  Design 

Traditionally  cost  estimation  has  involved  fixed-point  estimates:  how  long  the  project  will  take 
and  how  much  it  will  cost.  This  is  too  absolute  for  most  software  projects,  however,  as  noted  by 
Kennedy  and  colleagues  [Kennedy  2014].  They  propose  instead  a  set-based  approach,  in  which 
the  set  defines  a  range  of  acceptable  and  likely  outcomes.  IBM’s  Walker  Royce  has  espoused  sim¬ 
ilar  thinking  [Royce  2011].  The  developers  define  a  response  measure  (useful  for  testability)  for 
each  quality  attribute  and  then  find  the  limit  curve  of  the  subcomponent  of  interest  using  proto¬ 
type  spikes.  The  goal  is  to  learn  from  each  iteration  so  that  estimates  become  more  and  more 
tightly  bound. 

Quality  Attribute  Elicitation  Models 

Scenario-driven  design  refines  high-level  QARs  into  more  specific  scenarios.  In  a  Quality  Attrib¬ 
ute  Workshop,  scenarios  are  the  building  blocks  for  eliciting  feedback  on  a  set  of  architectural 
strategies  and  business  drivers  [Barbacci  2003].  A  scenario  is  described  as  specifically  as  neces¬ 
sary  to  exercise  the  desired  quality  attribute  (for  example,  a  login  use  case  for  the  security  attrib¬ 
ute).  Wood  describes  an  approach  that  assigns  scenarios  to  different  iterations  in  the  design  phase 
of  the  system  [Wood  2007].  The  collection  of  individual  scenarios  is  not  exhaustive,  however,  and 
more  coverage  would  be  necessary  to  fully  test  all  of  the  QARs. 

Architectural  tactics  are  design  decisions  that  influence  the  achievement  of  a  quality  attribute  re¬ 
sponse  [Bass  2012].  A  tactic  tree,  as  proposed  by  Bass  and  colleagues,  provides  a  rudimentary 
breakdown  of  such  decisions  for  common  quality  attributes  [Bass  2012].  These  are  likely  ap¬ 
proaches  to  decomposition.  For  example,  for  security  we  could  detect  attacks  or  resist  attacks,  and 
if  we  detect  attacks  we  have  a  number  of  design  choices,  including  detect  intrusions,  detect  denial 
of  service,  and  so  on. 

Patterns  bundle  design  decisions  (collections  of  tactics)  into  allocatable  units  to  optimize  the  solu¬ 
tion  according  to  industry  best  practices  for  addressing  common  problems.  Patterns  may  need  to 
be  broken  down  into  smaller  pieces  to  fit  in  an  iteration,  and  tactics  can  give  insight  into  their  de¬ 
composition. 

The  technique  described  by  Bass  and  colleagues  captures  QARs  as  six-part  scenarios  [Bass  2012]: 

1 .  source  of  stimulus 

2.  stimulus 

3.  environment 

4.  artifact 

5.  response 

6.  response  measure 
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This  technique  maps  to  the  “Given/When/Then”  behavior-driven  development  approach  (also 
called  “specification  by  example”)  [North  2006].  Parts  3  and  4  are  the  Given,  Parts  1  and  2  the 
When,  and  Parts  5  and  6  the  Then.  If  a  quality  attribute  is  amenable,  one  can  add  quality  thresh¬ 
olds  to  existing  stories  or  system  tests  and  monitor  outcomes  (response  measures).  This  serves  as 
the  Testable  property.  Value  and  cost  remain  outside  the  scope  of  a  quality  attribute  scenario.  This 
approach  is  particularly  suited  to  runtime  qualities  such  as  availability  and  performance,  which 
are  easily  operationalized.  Design  qualities  such  as  maintainability,  on  the  other  hand,  are  not  so 
simple  to  evaluate  quantitatively  (although  tools  are  improving  in  this  space).  Leffingwell  also 
notes  that  tests  may  vary  from  inspection,  to  special  harnesses,  to  continuous  monitoring  [Leff¬ 
ingwell  2011a].  Key  questions  to  investigate  are  whether  quality  attribute  scenarios  are  elaborated 
at  the  right  level  of  detail  to  allow  a  developer  to  complete  them  in  a  single  iteration  and  how  the 
QAR  can  be  suitably  operationalized  for  monitoring. 

Cost-Benefit  Analysis  and  Architecture  Improvement 

The  Cost  Benefit  Analysis  Method  (CBAM)  is  a  technique  for  evaluating  architectural  alterna¬ 
tives  using  stakeholder-driven  utility  curves  [Kazman  2002].  This  technique  is  primarily  a  means 
for  understanding  the  cost  and  value  of  an  architectural  approach  prior  to  undertaking  it  so  that  the 
development  team  works  on  the  high-value  activities.  The  utility  curve  shows  how  much  more 
benefit  moving  from  one  threshold  to  another  gives  for  how  much  cost.  However,  more  research 
is  needed  to  understand  the  incremental  cost  and  rework  associated  with  moving  along  the  curve 
for  given  design  fragments.  One  might  also  extend  the  concept  of  a  utility  curve  to  harness  real 
data  to  provide  iterative  monitoring  as  the  system  is  developed  to  support  measurement-driven 
analysis.  Related  techniques  in  real-options  analysis  economically  model  the  value  of  designs  and 
search-based  optimization  of  release  plans  [Sullivan  1999]. 

5.2  Software  Life-Cycle  Methodology 

Scaled  Agile  Framework 

Leffingwell’ s  SAFe  is  a  method  for  implementing  iterative  development  in  larger  software  pro¬ 
jects  (larger  size  usually  means  teams  with  more  than  10  to  15  people  collaborating  on  a  single 
project)  [Leffingwell  2014b].  There  is  well-developed  guidance  for  moving  from  high-level  “port¬ 
folio”  projects  to  team-sized  iteration  elements.  Requirements  flow  downstream  to  portfolio, 
product,  and  team  backlogs.  A  special  work  stream  handles  architecture-related  stories  (the 
QARs).  Architecture  epics  are  ongoing  cycles  for  refining  and  allocating  architecture -related  work 
to  team-sized  iterations  [Leffingwell  2011b,  2014a].  A  work-in-progress  limit  focuses  the  archi¬ 
tecture  team  on  a  limited  amount  of  work. 

Disciplined  Agile  Delivery 

Disciplined  Agile  Delivery  [Ambler  2012]  outlines  an  Inception  Phase  for  software  development. 
Similar  to  the  Rational  Unified  Process,  this  practice’s  Iteration  Zero  is  used  for  planning  the  iter¬ 
ations  involved  in  the  whole  development  life  cycle.  One  scopes  the  project  and  identifies  prelimi¬ 
nary  requirements,  possible  architectures,  and  unknowns  for  risk  management  and  prototypes.  As 
in  SAFe,  these  high-level  plans  are  then  handed  off  to  the  iteration  planning  exercise.  Throughout, 
the  development  team  focuses  on  risk  management  to  emphasize  the  value  being  delivered. 
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Incremental  Funding  Method 


In  the  book  Software  by  Numbers,  Denne  and  Cleland-Huang  introduce  the  Incremental  Funding 
Method  to  manage  software  development  projects  [Denne  2003].  Their  approach  is  primarily  an 
economic  one  and  centers  on  accurately  assessing  the  net  present  value  of  software  functionality. 
Functionality  is  refined  into  units  of  Minimally  Marketable  Features  (MMFs),  the  smallest  unit  a 
customer  would  value.  For  each  MMF,  a  refinement  is  proposed  that  highlights  common  architec¬ 
tural  constraints  for  all  MMFs.  A  dependency  map  then  outlines  the  possible  allocation  patterns 
for  building  the  MMF. 
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6  Related  Approaches 


There  are  several  areas  of  research  and  practice  that  we  do  not  probe  in  depth  but  that  may  have 
relevance,  either  in  terms  of  formal  tools  for  analysis  or  software  tooling  that  could  be  leveraged 
for  incremental  iterative  development  at  scale.  These  include  slicing  user  stories,  aspect-oriented 
software  development,  model  slicing,  valuation  approaches,  and  static  analysis. 

First  is  the  technique  of  slicing  user  stories  in  general,  not  only  nonfunctional  requirements.  For 
example,  as  explained  in  Gottesdiener  and  Gorman,  one  can  take  a  number  of  approaches  to  re¬ 
duce  any  high-level  user  story,  or  epic,  as  it  tends  to  be  called  [Gottesdiener  2010].  It  can  be  bro¬ 
ken  into  the  six  options  for  expansion  mentioned  in  Section  3.4 — user,  actions,  data,  business 
rules,  interfaces,  and  quality  attributes — to  create  a  much  larger  yet  better  described  epic.  All  of 
the  expansions  form  a  possible  option  portfolio  from  which  the  most  valuable  slices  can  be  chosen 
for  immediate  work. 

Aspect-oriented  software  development  (AOSD)  allows  quality  attributes  that  are  implemented  as 
behavior,  such  as  logging,  to  be  woven  into  existing  code  [Kiczales  1997].  The  idea  is  to  support 
crosscutting  concerns  in  the  language  itself.  Each  woven  requirement  could  serve  as  a  way  to  inte¬ 
grate  slices  into  program  development.  Adoption  of  the  AOSD  tooling  has  been  slow,  but  many 
properties  of  aspect-orientation  might  be  found  in  dependency  injection  and  configuration  ap¬ 
proaches  for  using  software  frameworks. 

Model  slicing,  well  described  by  Famelis  and  colleagues,  applies  formal  logic  to  the  problem  of 
specifying  a  range  of  possibilities  for  a  model-driven  development  [Famelis  2012].  The  develop¬ 
ment  team  describes  the  system  as  a  set  of  core  objects  and  “possibilities,”  which  represent  trajec¬ 
tories  that  the  system  might  take.  Model  slicing  is  similar  to  the  way  that  agile  methods  provide 
support  for  uncertainty,  only  here  the  support  is  backed  by  a  formalism  that  permits  analysis  of 
possible  futures.  Scalability  in  the  face  of  the  satisfiability  problem’s  assumed  intractability  is  a 
question,  of  course.  Similar  work  exists  in  the  requirements  engineering  community,  both  in 
search-based  optimization  [Zhang  2007]  and  in  requirements  “roadmaps.”  For  example,  Jureta 
and  colleagues  specify  possible  future  requirements  that  might  apply,  to  which  the  system  can  ei¬ 
ther  self-adapt  or  that  can  provide  a  trajectory  for  the  development  effort  (i.e.,  the  roadmap  speci¬ 
fies  the  tasks  that  ought  to  be  undertaken)  [Jureta  2010]. 

Underpinning  the  decomposition  of  a  story  is  the  need  for  prioritization  or  valuation  approaches 
that  assign  a  numeric  value  (possibly  ordinal,  possibly  ratio).  This  might  involve  a  simple 
weighting,  like  story  points,  or  more  complex  economic  approaches  that  use  real-options  analysis 
to  assess  the  net  present  value  of  the  quality  attribute  [Carriere  2010].  Economic  models  then  al¬ 
low  for  the  introduction  of  industrial  engineering  theories  for  scheduling,  such  as  weighted  short¬ 
est  job  first  [Reinertsen  2009].  Finally,  there  is  a  rich  literature  in  program  slicing  related  to  static 
analysis.  Here,  the  intent  is  to  understand  where  and  when  a  particular  object  of  interest,  typically 
a  variable,  is  accessed  or  accessible.  There  is  minimal  overlap  with  story  slicing,  but  some  formal 
approaches  may  be  useful.  For  example,  if  there  were  an  algebra  for  describing  architecturally  sig¬ 
nificant  requirements,  one  might  apply  slicing  operators  to  refine  that  variable  of  interest.  This  is 
the  approach  underlying  Khan  et  al.  [Khan  2008]. 
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7  Conclusion 


This  report  has  surveyed  the  current  state  of  the  art  in  refinement  and  allocation  of  QARs.  A  key 
finding  is  that  the  patterns  of  allocation  we  describe  do  not  exist  in  isolation  but  in  combinations. 
As  a  result,  it  is  no  surprise  that  no  one  technique  is  suitable  to  satisfy  key  QARs  that  are  relevant 
to  developing  business  capabilities.  Development  teams  practice  refinement  in  a  number  of  ways, 
but  ultimately  the  purpose  is  to  decompose  a  possibly  vague,  nonuniform  customer  requirement  or 
business  goal  into  iteration-sized  pieces.  In  the  allocation  process,  developers  then  take  those 
pieces  and  determine  when,  and  why,  to  work  on  them.  We  characterized  this  process  as  a  design 
activity:  refinement  and  allocation  are  explorations  of  the  problem  and  solution  spaces,  and  evolu¬ 
tionary,  iterative  development  allows  for  course  changes  when  the  development  team  acquires 
new  information.  Developers  should  work  toward  optimizing  the  satisfaction  of  QARs  in  the  con¬ 
text  of  the  cost  of  implementation,  cost  of  rework,  cost  of  delay,  tradeoffs  among  multiple  QARs, 
and  ultimate  value. 
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